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1.  Summary 

This  is  the  final  report  for  our  AFOSR  sponsored  project:  Information  Fusion  for  High  Level 
Situation  Assessment  and  Prediction.  Through  this  project,  we  developed  a  probabilistic 
framework  for  performing  high  level  information  fusion.  In  addition,  we  developed  algorithms  for 
performing  active  information  fusion  to  improve  both  fusion  accuracy  and  efficiency  so  that 
decision  making  and  situation  assessment  can  be  made  in  a  timely  and  efficient  manner.  Finally, 
we  applied  the  framework  and  the  algorithms  to  a  military  application  to  demonstrate  its  feasibility 
and  validity  for  high  level  information  fusion. 

In  this  report,  we  first  summarize  our  technical  accomplishments,  followed  by  a  discussion 
of  transitions  related  to  this  project.  The  related  paper  reprints  are  attached  with  this  report.  In 
addition,  the  latest  paper  reprints,  publications,  and  software  demos  from  this  project  may  also  be 
found  at  http://www.ecse.rpi.edu/~cvrl/lwh/ . 


2.  Introduction 

There  are  several  challenges  facing  information  fusion  for  situation  awareness  and  decision 
making  for  military  applications.  First,  sensory  data  generally  involves  multiple  data  types  such  as 
various  sensory  signals  (radars  either  at  the  same  or  different  frequencies  as  well  as  EO/IR  sensors), 
circumstantial  evidence,  geographical  information,  subjective  knowledge,  and  various  constraints. 
They  provide  information  at  different  levels  of  abstraction.  They  are  often  uncertain,  ambiguous, 
and  local.  Second,  for  many  military  domains,  the  world  situation  is  often  dynamic  and  unfolds 
over  time.  The  sensory  observations  also  evolve  over  time  to  reflect  changes  in  the  world.  To 
correctly  assess  and  interpret  the  world  situation,  an  adaptive  system  is  needed  that  can  reason  over 
time  since  it  is  often  the  temporal  changes  that  provide  crucial  information  about  what  we  try  to 
infer  and  understand.  Third,  many  military  applications  are  also  often  constrained  by  limited  time, 
resources,  and  complex  environments.  Given  a  vast  amount  of  sensory  data,  it  is  important  that 
fusion  be  carried  out  in  an  efficient  and  economical  manner  to  avoid  unnecessary  or  unproductive 
sensor  actions  and  computations  so  that  decision  about  the  situation  and  threat  can  be  made  in  a 
timely  and  efficient  manner.  A  high  level  fusion  system  that  is  dynamically  selective,  purposive 
and  sufficing  is  therefore  more  suitable.  This  can  be  achieved  through  active  fusion,  which 
involves  deliberate  sensor  management  to  achieve  efficient,  timely,  and  often  more  accurate 
information  fusion.  Specifically,  active  fusion  answers  the  following  questions:  what  information 
to  acquire  next  to  minimize  the  uncertainty  of  situation  assessment  and  to  maximize  the  overall 
expected  utility  of  decision  making,  what  sensors  can  be  used  to  acquire  the  information  efficiently, 
timely,  and  safely,  when  to  activate  the  sensors,  and  how  to  ftise  the  acquired  sensory  data 
efficiently.  The  goal  of  this  research  is  to  develop  a  high  level  multi-sensory  fusion  system  to 
address  all  these  challenges. 

Figure  1  illustrates  an  active  fusion  framework.  The  framework  consists  of  a  set  of 
information  sources  (sensors)  monitoring  the  world,  a  sensor  selection  mechanism,  a  fusion 
methodology,  the  results  of  fusion  (e.g.,  situation  understanding),  and  a  decision-making 
mechanism  for  identifying  a  set  of  actions  that  respond  to  the  fusion  results.  A  complete  active 
fusion  procedure  includes  deciding  a  sensor  set  that  achieves  the  optimal  trade-off  between  its  cost 
and  benefit,  activating  the  identified  sensors,  integrating  the  acquired  sensory  data,  and  making 
decisions  based  on  the  fusion  results. 
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Feedback 


Figure  1:  An  active  fusion  framework 

The  work  for  this  project  consists  of  three  parts:  The  first  part  focuses  on  developing  and 
implementing  a  probabilistic  framework  for  representing  and  integrating  sensory  data  of  different 
modalities  at  different  levels  of  abstraction.  One  component  of  this  part  of  the  work  is  the 
development  of  a  machine  learning  method  for  automatic  parameterization  of  the  framework  for  a 
particular  application.  The  second  part  is  to  develop  the  theories  and  algorithms  for  performing 
active  information  fusion.  This  requires  first  defining  a  sensor  selection  criterion,  developing 
computational  methods  to  efficiently  compute  the  criterion,  developing  algorithms  to  perform 
efficient  sensor  selection  using  the  criterion,  and  finally  developing  methods  to  efficiently  fuse  the 
acquire  the  sensory  data.  The  third  part  of  this  research  is  to  demonstrate  the  proposed  framework 
and  algorithms  for  a  military  application  with  a  prototype  software.  We  have  made  significant 
progress  in  each  of  these  three  areas  as  detailed  below. 

3.  An  Unified  Probabilistic  Framework  for  High  Level  Information  Fusion 

Knowledge  in  a  military  domain  is  complex,  uncertain,  and  dynamic.  We  need  a  framework 
that  can  accurately  model  the  incomplete/uncertain  world  knowledge  and  that  provides  a 
mechanism  for  systematically  and  actively  integrating  dynamic  information  from  disparate  sources. 

The  Dynamic  Bayesian  Networks  (DBNs)  appear  ideal  for  meeting  these  requirements.  DBNs  are 
probabilistic  graphs  with  nodes  representing  random  variables  and  links  representing  the  casual 
relationships  among  the  connected  nodes.  Rooted  firmly  in  the  long-established  field  of  probability 
theory,  DBNs  can  systematically  and  actively  combine  corrupted  sensory  inputs  of  different 
modalities  occurring  over  different  time  frames  to  produce  a  consistent,  coherent,  and  accurate 
global  picture  of  the  underlying  events  and  to  generate  appropriate  response  recommendations. 
Knowledge  modeling  using  DBNs  involves  identification  and  representation  of  the  sensory 
information,  contextual  information,  hypotheses  (what  we  want  to  infer),  and  their  dependencies  as 
well  as  their  uncertainties  and  dynamics. 

Specifically,  DBNs  provide  a  coherent  and  unified  hierarchical  probabilistic  framework  for 
representing  and  integrating  information  from  diverse  and  correlated  sources,  including  sonar, 
radar,  images,  tactical  information,  and  knowledge  expressed  by  domain  experts  or  synthesized 
through  discovery  techniques.  Furthermore,  DBNs  dynamically  evolve  and  grow  to  accommodate 
the  new  happenings  and  to  assess  the  current  situation  not  only  based  on  the  current  information  but 
also  utilize  information  produced  during  previous  time  frames,  as  alternative  scenarios  are 
reinforced  or  ruled  out  dynamically.  Figure  2  shows  an  example  of  using  DBN  to  integrate  various 
sensory  data  to  infer  enemy  intent.  Details  about  this  work  may  be  found  in 

[Zhang&Ji06a,Liao&Ji07b]. 
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Figure  2:  Generic  structure  of  a  slice  of  a  DBN  for  hierarchically  representing  knowledge  related  to 
inferring  enemy  intent.  Rectangular  nodes  represent  sensors  that  produce  the  sensory  data  at  different  levels. 

The  benefit  with  DBN  for  knowledge  representation  is  that  the  representation  is  in  a 
hierarchical  structure.  The  hierarchical  knowledge  representation  corresponds  nicely  to  different 
levels  of  fusion  according  to  the  Joint  Directors  of  Laboratories  (JDL)  Data  Fusion  Model 
classification  [SteinbergBowmanOl],  For  example,  the  target  node  at  the  top  stores  the  results  of 
level-3  fusion  while  the  content  of  the  leaf  nodes  corresponds  to  the  level-1  fusion.  The 
intermediate  hidden  nodes  correspond  to  results  of  level-2  fusion.  Specifically,  the  level-1  nodes  in 
the  bottom  are  used  to  extract  sensory  measurements/features  from  objects  of  interest.  The  level-2 
nodes  in  the  middle  fuse  the  spatial  and  temporal  relationships  between  entities  to  recognize 
objects,  their  relationships,  and  their  movement  tracks.  Finally,  the  level-3  nodes  at  the  top 
perform  situation  assessment,  enemy  intent  prediction,  or  threat  assessment.  Level-3  fuses  the 
combined  activity  and  capability  of  enemy  forces  to  infer  their  intention  and  to  assess  the  threat 
they  pose. 

While  excellent  in  knowledge  and  uncertainty  representation,  DBNs  do  not  have  the 
decision  making  capability.  Decision  such  as  what  sensory  actions  to  take  next  and  what  course  of 
actions  to  choose  need  be  made  outside  the  DBN  framework,  usually  in  a  heuristic  manner  and 
independent  of  the  fusion  process.  To  overcome  this  limitation,  we  extend  the  framework  from  the 
DBNs  to  Dynamic  Influence  Diagrams  (DIDs).  Like  Bayesian  networks,  DIDs  can  represent  both 
static  and  dynamic  uncertain  knowledge  in  a  hierarchical  graphical  structure,  with  nodes 
representing  the  random  variables  and  directed  links  between  nodes  representing  the  casual 
relationships.  Unlike  DBNs,  a  DID  also  includes  decision  nodes  that  may  be  used  to  explicitly 
represent  various  actions  to  take  in  response  to  the  results  of  information  fusion.  Decision  and 
utility  theories  can  then  be  used  within  the  DID  framework  to  determine  the  optimal  actions  to  take. 
This  feature  is  especially  attractive  for  active  sensing  since  DID  provides  a  mechanism  to 
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systematically  connect  decision-making  with  sensor  management,  i.e.,  allowing  sensory 
management  in  response  to  decision  making. 

In  summary,  DIDs  provide  a  unified  mathematical  framework  for  simultaneously  modeling 
and  integrating  sensory  data  selection,  sensory  data  fusion,  situation  assessment,  and  decision 
making.  Such  a  model  yields  several  advantages.  First,  it  provides  a  coherent  and  fully  unified 
hierarchical  probabilistic  framework  for  representation  and  integration  of  sensory  data  of  different 
modalities  at  different  levels  of  abstraction,  and  for  decision  making  via  inference  under 
uncertainty.  Second,  it  embeds  both  the  sensors'  contribution  to  decision-making  and  their 
operating  cost  in  one  framework  to  allow  systematic  determination  of  an  optimal  sensor  subset 
based  on  utility  theory,  probability  theory  and  information  theory.  Third,  it  systematically 
incorporates  the  evolution  of  the  situation  and  sensory  data  as  well  as  accounts  for  the  temporal 
aspect  of  decision  making  with  a  dynamic  structure.  Under  the  framework,  timely  and  effective 
decision  can  be  made  by  dynamic  inference  based  on  selecting  a  subset  of  sensors  with  the  optimal 
trade-off  between  their  cost  and  benefit.  Figure  3  shows  an  example  of  DID  framework  for 
modeling  space  situation  awareness  and  space  threat  mitigation.  Additional  examples  about  DIDs 
may  be  found  in  [Liao&Ji07a,  Liao&Ji06b] 


Figure  3:  An  example  influence  diagram  for  modeling  space  threat  assessment  and  mitigation, 
where  the  circular  nodes  represent  the  random  variables  (like  the  nodes  in  DBN),  the  rectangular 
nodes  represent  the  action/decision  nodes  (e.g.  sensory  action  and  course  of  action),  and  the 
diamond  nodes  represent  the  utility  of  the  actions/decisions. 
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4  .  Model  Construction  and  Learning  with  both  Quantitative  and  Qualitative  Knowledge 

Given  the  generic  DBN/DID  as  discussed  in  the  previous  section,  a  key  problem  with 
graphical  modeling  is  to  determine  the  model  topology  (the  model  structure)  and  its  parameters  for 
a  particular  application.  Model  topology  determination  requires  identifying  the  appropriate  random 
variables  and  their  relationships.  Model  parameterization  involves  quantifying  each  root  node  (the 
topmost  node)  with  a  prior  probability  and  each  link  with  a  conditional  probability  table  (CPT) 
describing  the  conditional  probability  of  the  node  given  all  possible  outcomes  for  its  parent(s).  For 
a  dynamic  ID,  we  also  need  to  specify  relationships  between  two  neighboring  static  slices  with  a 
transitional  probability  matrix. 

So  far,  we  have  been  using  domain  expert  to  help  construct  the  model  and  to  provide  some 
initial  estimation  of  the  model  parameters.  To  further  improve  the  model  parameters,  we  developed 
a  new  machine  learning  method  [Liao&etal07]  that  refines  the  initial  model  parameters  by 
combining  the  subjective  knowledge  with  some  training  data.  Study  [Johansson&Falkman06] 
shows  for  many  military  applications  training  data  are  hard  to  acquire  due  to  various  reasons 
including  security  or  lack  of  data.  On  the  other  hand,  it  is  shown  that  for  a  given  application,  a 
number  of  general  parameters  can  often  be  extracted  from  the  interview  with  the  officers  and  from 
the  discussion  with  the  military  experts.  This  subjective  knowledge  may  often  present  qualitatively 
instead  of  quantitatively  in  the  form  of  relationships  among  some  parameters  of  the  model  instead 
of  all  parameters.  Two  types  of  domain  specific  knowledge  we  use  in  our  method  are  the  range  of 
some  parameters  and  the  qualitative  relationships  among  some  parameters.  This  kind  of  subjective 
and  prior  knowledge  is  very  different  from  the  conventionally  used  quantitative  knowledge  such  as 
prior  probability  distribution  on  all  parameters.  The  latter  is  often  hard  to  quantify  or  requires 
strong  assumptions  to  quantify.  The  qualitative  knowledge,  on  the  other  hand,  is  often  ignored  due 
to  the  difficulty  in  incorporating  them  into  the  model.  To  utilize  the  qualitative  knowledge,  we 
developed  a  method  that  estimates  the  parameters  of  a  Bayesian  network  under  incomplete/sparse 
training  data  but  with  some  subjective  constraints.  Specifically,  we  introduce  a  modified 
expectation  maximization  (EM)  method  that  performs  parameter  estimation  in  two  steps:  the  E-step 
and  the  M-step.  The  E-step  allows  to  estimate  the  missing  data  based  on  the  available  data  while 
the  M-step  allows  to  estimate  the  model  parameters.  The  EM  algorithm  is  modified  as  follows.  For 
the  M  step,  it  is  modified  to  incorporate  qualitative  prior  knowledge  about  the  parameters  of  the 
model.  For  example,  the  domain  knowledge  about  the  range  of  the  parameters  and  the  qualitative 
relationships  among  the  parameters  may  be  incorporated  in  the  M  step  to  modify  the  objective 
function  and  to  constrain  the  estimated  parameters  within  a  scope. 

Experiments  with  both  synthetic  and  real  data  demonstrate  significant  improvement  of  the 
proposed  method  for  estimating  parameters  of  the  Bayesian  network  under  incomplete  or  sparse 
training  data.  Specifically,  Figure  4  gives  an  example  of  a  Bayesian  Network,  where  the  shaded 
nodes  represent  variables  that  do  not  have  training  data.  Instead,  some  quantitative  constraints  are 
available  for  those  nodes. 
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Figure  4:  An  example  Bayesian  network,  with  insufficient  training  data.  Data  are  not  available  for 
the  shaded  nodes.  Some  qualitative  subjective  knowledge  is  available  on  those  and  other  nodes. 


We  apply  the  modified  EM  method  to  estimate  the  parameters  of  the  BN  using  the  incomplete  data 
and  the  qualitative  constraints.  The  results  are  shown  in  Figure  5,  where  our  modified  EM  method 
is  compared  with  the  results  of  the  conventional  EM  method.  We  can  see  a  significant 
improvement  in  the  estimation  accuracy  in  terms  of  the  KL  divergence. 


Figure  5:  Results  of  parameter  estimation  using  the  proposed  constrained  EM  (CEM)  v.s.  the 
conventional  EM. 

Additional  details  about  the  CEM  method  may  be  found  in  [Liao&etal07]. 


5.  Active  Information  Fusion 

Given  the  probabilistic  frameworks  discussed  above,  information  fusion  can  be  carried  out 
through  a  probabilistic  inference  by  systematically  propagating  the  impacts  of  the  received  sensory 
data  through  the  model.  Belief  propagation  in  a  DID  framework  is  usually  time-consuming.  Given 
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often  a  vast  amount  of  sensory  data  as  well  as  the  constraints  on  resource  and  time,  it  is  important 
to  perform  information  fusion  in  a  selective  and  purposive  manner  so  that  decision  can  be  made 
quickly  and  economically.  This  can  be  achieved  through  active  information  fusion,  which  involves 
deliberate  sensor  management  to  achieve  efficient,  timely,  and  often  more  accurate  information 
fusion. 

As  shown  in  Figure  6,  the  two  main  tasks  for  active  fusion  are  sensor  selection  and  sensory 
data  fusion  through  belief  propagation.  Specifically,  sensor  selection  is  to  decide  which  subset  of 
sensory  observations  should  be  acquired  next.  Given  the  identified  sensory  observations,  the 
corresponding  sensors  are  activated  to  acquire  the  data,  and  the  acquired  data  are  subsequently 
integrated.  Sensor  selection  includes  two  steps.  The  first  step  is  achieved  by  designing  a  sensor 
selection  criterion  that  represents  the  trade-off  between  the  sensor  benefit  and  sensor  cost.  This  is 
then  followed  by  the  second  step,  which  searches  the  sensor  space  using  the  criterion  to  identify  the 
optimal  sensor  subset.  There  are  significant  computational  difficulties  associated  with  computing 
sensor  selection  criterion  and  searching  for  the  optimal  sensor  subset  using  the  criterion.  In  the 
following  sections,  we  will  discuss  methods  to  address  the  two  computational  difficulties. 


Figure  6:  the  steps  of  active  fusion  for  decision  making 

5.1  Efficient  Sensor  Selection  Criterion  Computation 

We  have  studied  two  sensor  selection  criteria:  mutual  information  (MI)  and  value  of 
information  (VOI).  MI  defines  the  potential  of  a  set  of  sensors  in  reducing  the  uncertainty  of  the 
current  hypothesis  about  the  situation.  VOI,  on  the  other  hand,  is  defined  as  the  difference  of  the 
maximum  expected  utilities  with  and  without  the  selected  information  source  (s)  with  respect  to 
decision  making.  As  a  decision-theoretic  criterion,  VOI  is  generally  different  from  the  information- 
theoretic  measures  such  as  mutual  information  since  VOI  directly  relates  sensor/resource  allocation 
(e.g.  assignment  of  a  reconnaissance  operation)  to  decision-making  (e.g,  taking  defensive/offensive 
actions),  while  information  theoretic  criterion  relates  sensor  management  to  situation  awareness. 
MI  is  therefore  more  suitable  for  situation  awareness  while  VOI  is  more  suitable  for  decision 
making  in  response  to  situation  awareness.  For  this  research,  both  MI  and  VOI  can  be  used  to  rate 
the  usefulness  of  various  sensory  sources  with  respect  to  the  decision  making,  and  to  decide 
whether  pieces  of  evidences  are  worth  acquisition  before  actually  activating  the  sensors. 
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Exact  computation  of  either  MI  or  VOI  for  a  set  of  sensors  is  expensive  and  is  often  NP- 
hard,  which  is  practically  infeasible.  One  common  solution  to  this  problem  is  to  use  myopic 
approach,  i.e.,  selecting  one  optimal  sensor  at  a  time.  Obviously,  such  selection  is  not  always 
reasonable  since  a  decision  maker  will  often  not  act  after  acquiring  data  from  only  one  information 
source.  Instead,  a  decision  maker  often  needs  to  collect  multiple  pieces  of  evidence  from  multiple 
sources  before  making  a  decision.  Therefore,  it  is  necessary  to  compute  non-myopic  MI  and  VOI. 
Hence,  an  approximate  computation  of  non-myopic  VOI  and  MI  is  necessary  to  make  its 
computation  feasible  for  practical  applications. 

In  [Zhang&Ji05],  we  introduce  a  graph-theoretic  approach  for  efficiently  approximating  the 
mutual  information  for  a  set  of  sensors.  Specifically,  we  propose  a  new  quantitative  measure  for 
sensor  synergy,  based  on  which  a  sensor  synergy  graph  is  constructed.  Using  the  sensor  synergy 
graph,  we  introduce  an  alternative  measure  (the  least  upper  bound  of  mutual  information)  to 
approximate  multi-sensor  mutual  information  for  characterizing  sensor  information  gain.  Studies 
show  that  the  new  measure  is  very  close  to  mutual  information  in  value  yet  can  be  very  efficiently 
computed.  Figure  7  shows  the  closeness  between  the  least  (greatest)  upper  bound  of  mutual 
information  and  the  exact  mutual  information  for  sensor  subsets  of  different  sizes.  It  is  clear  that 
the  proposed  criterion  based  on  the  least  upper  bound  of  mutual  information  closely  follows  the 
values  of  the  exact  mutual  information. 


Figure  7:  the  closeness  between  the  least  (greatest)  upper  bound  of  mutual  information  and  the 
exact  mutual  information  for  sensor  subsets  of  different  sizes. 

Computationwise,  since  the  substitute  measure  can  be  written  simply  as  the  sum  of  the  mutual 
information  of  only  pairwise  sensors  and  singleton  sensors,  its  computation  cost  is  relatively  very 
low.  Therefore,  the  computational  difficulty  in  computing  exactly  higher  order  mutual  information 
can  be  circumvented  by  computing  only  the  least  upper  bounds  of  mutual  information.  An 
extensive  simulation  study  demonstrates  both  the  optimality  and  efficiency  of  the  sensor  selection 
criterion.  Details  may  be  found  in  [Zhang&Ji05,  Zhang&Ji07] 

In  [Liao&Ji06],  we  introduced  an  algorithm  to  approximately  compute  non-myopic  VOI 
efficiently  by  exploiting  the  central-limit  theorem  as  well  as  by  exploiting  the  statistical 
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dependencies  among  the  sensors.  Specifically,  computing  VOI  requires  a  sum  over  all  possible 
combinations  of  sensory  observations,  which  is  time  consuming  for  a  large  number  of  sensors  or  for 
sensors  with  many  states.  In  our  method,  we  treat  each  term  in  the  summation  as  a  random 
variable,  whose  mean  and  standard  deviation  can  be  individually  estimated.  The  central  limit 
theorem  can  then  be  used  to  approximate  the  mean  and  standard  deviation  of  the  sum,  based  on 
which  VOI  can  be  efficiently  approximated.  Additional  computational  saving  is  achieved  by 
exploiting  the  statistical  dependencies  among  sensors  when  computing  mutual  information.  Studies 
with  both  synthetic  and  real  data  show  that  the  approximated  method  for  computing  VOI  produces 
significant  improvement  in  computation  speed  with  minimum  loss  in  estimation  accuracy.  Figure 
8  shows  results  from  a  simulation  study,  where  it  shows  the  comparison  of  the  proposed 
approximated  method  for  computing  VOI  against  the  exact  VOI  computation  method  in  terms  of 
accuracy  and  speed  for  sensor  subsets  of  different  sizes.  It  quantitatively  demonstrates  the 
significant  improvement  in  computational  efficiency  with  minimum  loss  in  accuracy,  especially  for 
sensor  sets  with  many  sensors. 


3  4  5  6  7  8 

Size  of  o  Set 


9  10  11 


(a) 


Figure  8:  Comparison  of  the  exact  computation  of  VOI  v.s.  the  proposed  approximation  method  for 
sensor  subset  of  different  sizes  in  terms  of  accuracy  (a)  and  time  (b). 


5.2  Efficient  Sensor  Selection 

Given  the  criterion  for  sensor  selection,  the  selection  of  a  sensor  subset  from  several  sensor 
candidates  can  be  regarded  as  a  search  problem  in  a  combinatorial  space.  The  goal  is  to  find  the 
most  appropriate  solution  among  all  possible  sensor  combinations.  Since  the  number  of  sensor 
combinations  2k  increases  exponentially  with  the  number  of  sensors  k,  it  is  infeasible  for  an 
exhaustive  search  with  even  a  moderate  k.  Various  methods  have  been  proposed  to  perform  search 
efficiently  but  often  with  suboptimal  solution.  These  methods  include  the  greedy  approach  (select 
the  top  K  sensors),  the  myopic  approach  (select  one  sensor  at  a  time),  and  the  heuristic  methods. 

We  have  attacked  the  sensor  selection  problems  from  two  different  aspects.  First,  we  study  the 
properties  of  sensor  selection  criteria  under  different  conditions  so  that  a  greedy  approach  can 
produce  performance  guarantee.  Specifically,  our  recent  research  proves  that  information-theoretic 
sensor  selection  criterion  such  as  mutual  information  has  the  submodular  property  under  certain 
conditions  and  given  such  a  property,  the  greedy  sensor  selection  method  guarantees  the  near- 
optimal  sensor  selection  accuracy  in  polynomial  time  [Liao&Ji07].  Using  this  special  property,  we 
developed  a  greedy  sensor  selection  method  based  on  mutual  information  that  can  efficiently 
identify  the  near  optimal  subset  of  sensors.  Table  1  shows  results  of  the  proposed  sensor  selection 
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methods  (algla,  alglb,  and  alg2)  against  the  conventional  greedy  approach  in  terms  of  accuracy 
and  speed. 

Table  1:  Comparison  of  the  proposed  sensor  selection  methods  v.s.  the  conventional  greedy 
sensor  selection  method 


Alg.  la 

Alg.  lb 

Alg.  2 

Greedy 

Error  Rate 

0.06 

0.06 

0.10 

0.22 

Information  gain  rate 

0.99 

0.99 

0.97 

0.91 

Running  time  rate 

0.35 

0.10 

0.06 

0.03 

In  table  1,  the  error  rate  is  the  ratio  between  the  number  of  mis-selection  cases  and  the  overall 
number  of  cases,  where  a  mis-selection  case  is  defined  as  the  case  that  the  selected  sensor  set  has 
more  than  one  sensor  different  from  the  ground-truth.  The  information  gain  rate  is  the  ratio 
between  the  mutual  information  of  the  selected  sensor  set  and  that  of  the  optimal  sensor  set.  The 
running  time  rate  is  the  ratio  between  the  running  time  of  the  proposed  methods  and  that  of  the 
brute-force,  averaged  over  500  testing  cases.  It  is  apparent  that  the  proposed  sensor  selection 
methods  can  produce  more  accurate  results  with  little  loss  in  computational  speed.  Additional 
information  about  this  work  may  be  found  in  [Liao&Ji07b]. 

Second,  we  also  introduced  a  graphic-theoretic  approach  [Zhang&Ji05]  to  efficiently  find  the 
near  optimal  subset  of  sensors.  The  method  first  defines  a  sensor  synergy  measure,  based  on  which 
a  synergy  graph  is  constructed  to  represent  synergy  among  sensors.  The  graph  is  subsequently  used 
to  prune  the  sensor  space  so  that  a  large  set  of  less  synergetic  sensor  combinations  can  be 
eliminated,  significantly  reducing  sensor  space  and  as  a  result,  the  optimal  subset  can  be  more 
efficiently  identified  in  the  reduced  sensor  space.  Table  2  summarizes  the  results  of  comparison  of 
the  proposed  sensor  selection  method  with  the  random  selection  method  and  with  the  brute  force 
method  in  both  accuracy  and  time  for  sensor  subsets  of  different  sizes. 

Table  2  Comparison  of  the  proposed  method,  with  the  random  method  and  the  brute  force  method. 


Number 

i  Our  Approach 

Random  Method 

Brute-Force 

of 

Relative  utility  difference 

Run  time 

Relative  utility  difference 

Run  time 

Sensors 

of  our  method  to  brutal  force  methods 

(Seconds) 

of  random  method  to  brutal  force  methods 

(Seconds) 

7 

1.56% 

1 .020 

21.1 .5% 

63.87 

8 

1.77% 

1.099 

28.32% 

355.05 

9 

2.75% 

1.209 

36.54% 

2967.36 

10 

1.89% 

1 .430 

39.19% 

13560.54 

It  is  clear  from  the  table  that  compared  with  the  random  sensor  selection,  the  proposed  sensor 
selection  method  improves  accuracy  significantly.  Against  the  brute  force  method,  the  speed 
improvement  is  several  orders  of  magnitude  with  minimum  loss  in  accuracy.  More  about  this 
work  may  be  found  in  [Zhang&Ji07]. 
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6.  Efficient  Sensory  Data  Fusion 

To  fuse  the  information  collected  from  the  sensors  for  timely  decision-making,  an  efficient 
belief  propagation  algorithm  is  needed  since  typical  belief  propagation  based  on  ID/BN  is  NP-hard. 
We  developed  a  factorization  tree  inference  (FTI)  algorithm  [Liao&Ji04,  Zhang&Ji06b]  to 
efficiently  integrate  the  acquired  sensory  data  and  to  update  the  hypothesis  belief.  The  algorithm 
factors  out  the  computations  that  are  common  for  sequential  inference  so  that  they  can  be  computed 
only  once,  and  can  be  applied  to  the  subsequent  inference  without  re-computing  them,  therefore 
significantly  reducing  the  sensory  fusion  time.  Details  about  these  algorithms  may  be  found  in 
[Liao&Ji04,  Zhang&Ji06b].  Figure  9  compares  the  inference  efficiency  of  the  factor  tree  approach 
with  other  popular  inference  methods.  It  is  clear  that  the  proposed  factor  free  is  better  for 
sequential  inference. 


Figure  9:  Comparison  of  factor  tree  approach  with  other  inference  methods  (Variable 
Elimination  (VE)  and  Clique  Tree  Propagation  (CTP))  for  sequential  inference 

7.  An  Application  Example 

The  proposed  framework  may  be  used  for  modeling  complex  dynamic  events,  which  requires 
efficient  interpretation  of  data  from  different  sources  to  achieve  high  level  understanding  of  the 
causes  that  lead  to  the  observed  data  as  well  as  to  take  appropriate  course  of  actions  to  alleviate  the 
threat  the  situation  may  pose.  Here,  we  briefly  demonstrate  the  application  of  the  proposed 
framework  for  a  multi-stage  battlefield  situation  assessment. 

The  scenario  develops  during  a  period  of  growing  hostility  between  the  nation  A  (Blue 
force)  and  the  nation  C  (Red  force).  The  island  Mz  locates  between  the  two  nations  and  it  was 
occupied  by  the  Blue  force  since  the  World  War  II.  The  Red  force  wants  to  seize  the  island  back. 
As  the  situation  worsens,  the  Blue  force  designates  an  area  covering  land  and  sea  as  a  restricted 
zone,  and  declares  any  activities  in  the  restricted  area  as  hostile. 

To  monitor  the  situation,  the  Blue  force  surveillance  facilities  include  a  number  of  shore 
sensors,  unmanned  aerial  vehicles  (UAVs),  surveillance  helicopters  (RAH66  Comanche),  etc..  As 
hostilities  break  out  openly,  the  Red  force  may  want  to  destroy  the  surveillance  facilities  used  by 
the  Blue  force.  Further,  the  Red  force  may  also  pursue  air  strike  or  surface  attack  of  the  Blue  force 
if  necessary. 
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Using  its  assets,  the  Blue  force  monitors  the  activities  of  the  Red  force,  infers  its  intent,  and 
takes  appropriate  actions  to  mitigate  any  threat  the  Red  force  may  pose.  A  dynamic  influence 
diagram  as  shown  in  Figure  10  is  constructed  to  help  the  Blue  force  to  assess  the  intent  of  the  Red 
force  and  to  determine  the  best  course  of  actions.  A  set  of  hypotheses  representing  possible  Red 
force  intentions  include:  1)  Passive:  monitor  the  Blue  forces  in  the  restricted  zone;  2)  Defensive: 
conduct  active  reconnaissance  and  maintain  a  defensive  presence;  3)  Offensive:  mount  naval  attack 
or  infantry  artillery  engagement  (surface-to-air  or  surface-to-surface  attack)  on  the  Blue  forces  with 
the  intention  of  destroying  the  Blue  forces  as  well  as  their  offshore  surveillance  facilities. 
Corresponding  to  each  enemy  intent,  the  Blue  force  may  take  different  actions. 


Figure  10:  A  dynamic  influence  diagram  model  for  the  battlefield  scenario.  Si-Sio  represent  the 
surveillance  facilities  the  Blue  force  uses  to  collect  sensory  data  on  Red  force  activities. 

Given  the  model,  at  each  time  instant,  we  then  apply  the  active  fusion  algorithms  to  efficiently 
identify  the  optimal  sensors  to  use  in  order  to  timely  and  efficiently  identify  enemy  intent,  based  on 
which  we  then  decide  the  optimal  action  the  Blue  force  can  take  in  order  to  minimize  any  threat 
from  the  Red  force.  Figure  1 1  provides  the  results  of  inference  over  time  about  the  intent  of  the 
Blue  force,  the  sensors  used  by  the  Red  force,  and  the  actions  taken  to  counter  any  threat  from  the 
Red  force. 
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Assessment 

Stage 

Probability  of 
Hypothesis 

Blue  Force 

Action  Taken 

Sensors 

Selected 

1 

P(Pas)=0.3333 
P(Def)  0.3333 
P(Oflf)=0.3333 

Observe 

S4,  Sr, ,  So 

Shu  S7 

Si  2 

:  2 

P(Pas)=0.7379 
P(Def)  0.1367 

P(  Off  )=0.1253 

Further 

Observe 

S'9,  Sjo 

3 

P(Pas)=0.7465 

P( Defy  0.1327 
P(Off)=().  1 208 

Interception 
( Red  Force  may 
Change  Intention) 

4 

P(Pas)=0.2590 
P(Def)=0.55 1 8 
P(Off)=().  1 892 

Further 

Observe 

Sr,  >  S7 

So,  S'io 

.  5 

P(Pas)=0.2 164 
P(Def)=0.6877 
P(0ff)=0.0959 

Further 

Observe 

Sr,.  S7 

S’s,  S'n 
:  Si  2 

6 

P(  Pas)=0.02 1 3 
P(Def )=0.838 1 
P(0ff)=0.1406 

Minor  Offensive 

(Red  Force  May 

C  hange  intention) 

Figure  1 1  Step  by  step  inference  results  for  the  example 
Details  about  this  example  may  be  found  in  [Zhang  &  Ji05,Liao&Ji07b] . 

8.  Active  Information  Fusion  Demo  Software 

Besides  the  application,  we  have  also  developed  a  real-time  prototype  software  to 
demonstrate  the  proposed  methods.  In  the  demo  software,  several  sensor  selection  algorithms 
discussed  in  section  5.2  are  implemented  including  the  greedy  approach,  the  graph-theoretic 
approach,  the  myopic  approach,  the  random  approach,  and  the  brutal  force  approach.  User  can 
change  the  model  parameters,  the  reliability  and  cost  for  each  sensor.  Then,  based  on  the  inputs,  the 
system  returns  the  selected  sensors  by  using  different  sensor  selection  algorithms.  The  returned 
results  are  displayed  so  that  the  user  can  compare  the  efficiency  and  accuracy  of  different 
algorithms.  We  have  two  versions  of  the  demo  software:  one  is  based  on  the  graph-theoretic 
approach  that  uses  only  mutual  information  as  the  selection  criterion  and  the  other  is  based  on  the 
greedy  sensor  selection  approach  that  uses  either  mutual  information  or  value  of  information  as  the 
selection  criteria.  Figure  12  shows  the  interface  for  the  software  based  on  the  graph-theoretic 
approach.  Figure  13  shows  the  interface  for  the  software  based  on  the  greedy  approach. 
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Figure  12:  The  active  fusion  demo  software  interface  that  uses  mutual  information  as  sensor 
selection  criterion  and  uses  the  graph-theoretic  approach  for  sensor  selection. 
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Figure  13:  The  active  fusion  demo  software  interface  that  uses  either  mutual  information  or  value 
of  information  as  sensor  selection  criteria  and  uses  the  greedy  approach  for  sensor  selection. 
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Both  software  are  available  to  the  government  upon  request. 

9.  Interaction/Transitions 

Over  the  past  few  years,  the  PI  actively  interacted  with  the  AFRL-Rome  lab  to  identify  new 
applications  for  the  proposed  theories  and  algorithms.  Specifically,  the  proposed  active  fusion 
strategy  has  been  applied  to  addressing  the  computational  complexity  problem  plaguing  AFRL- 
Rome’ s  Causal  Analysis  Tool  (CAT)  for  military  plan  assessment,  leading  to  a  significant 
computational  saving  for  military  plan  evaluation  [Zhang&Ji06b]. 

Finally,  through  this  project,  we  have  provided  complete  financial  support  for  one  Ph.D 
student,  partial  support  for  another  Ph.D  student,  and  financial  support  for  two  postdoctoral 
associates.  In  particular,  one  Ph.D  student  (Ms.  Wenhui  Liao)  received  her  Ph.D  in  December, 
2006  with  complete  financial  support  from  this  grant.  Her  dissertation  is  on  high  level  active 
information  fusion. 

10.  Conclusion 

In  summary,  through  this  research  we  made  several  contributions  as  summarized  below: 

1)  Developed  and  implemented  a  unified  probabilistic  framework  for  simultaneously  modeling 
sensory  data,  sensor  selection,  situation  assessment,  and  decision  making.  Specifically,  we 
studied  and  implemented  two  frameworks:  Dynamic  Bayesian  Networks  and  Dynamic 
Influence  Diagrams. 

2)  Developed  and  implemented  a  learning  algorithm  to  estimate  model  parameters  when 
data  are  incomplete  using  domain  specific  qualitative  knowledge.  Specifically,  we 
developed  a  constrained  Expectation-Maximization  learning  algorithm  for  learning  the 
parameters  of  the  Bayesian  network  under  incomplete/sparse  training  data  with  qualitative 
constraints. 

3)  Introduced  sensor  selection  criteria  and  develop  computational  methods  to  efficiently 
compute  the  criterion.  We  studied  two  sensor  selection  criteria  (mutual  information  and 
value  of  information)  and  developed  methods  to  efficiently  compute  them 

4)  Developed  active  sensor  selection  methods  to  efficiently  identify  a  subset  of  optimal  sensors 

a.  a  greedy  approach  based  on  submodular  property  of  mutual  information  with 
performance  guarantee 

b.  a  graph-theoretic  approach  to  systematically  estimate  synergy  among  sensors  and  to 
significantly  reduce  the  sensor  search  space  by  eliminating  a  large  number  of 
unpromising  sensor  combination  through  a  synergy  graph. 

5)  Developed  a  belief  propagation  method  to  efficiently  integrate  the  acquired  sensory  data 
and  update  the  belief  to  current  hypothesis.  Specifically,  we  introduced  a  factorization 
method  for  efficient  sequential  inference  in  both  BN  and  ID. 

6)  Demonstrated  the  feasibility  of  the  framework  and  algorithms  with  realistic  military 
applications.  Developed  user-friendly  prototype  software  for  this  purpose. 

7)  Worked  with  AFRL/Rome  lab  to  apply  the  proposed  active  fusion  methods  to  address  the 
military  plan  assessment  problem. 

8)  Produced  one  Ph.D  dissertation  and  10  publications  in  the  referred  journals  and 
conferences. 
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